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Abstract 

This paper describes a project which has the objective of evaluating Web resources for 
language learning. Issues of quality of Web resources, the need for an evaluation procedure 
and the difficulties in implementing it are discussed first. The development of criteria on which 
to base such an evaluation is then outlined, followed by a description of its implementation on 
a Web page which indexes Spanish resources. 



1. Introduction 

Web resources for language leaming/teaching have started to proliferate at an extraordinary rate. 
Some of these resources are specifically designed for language learning or teaching, e.g. tutorials, 
reading comprehension exercises, grammar references, dictionaries etc. Others are seen as resources 
for language leaming/teaching exploitation, such as newspapers, radio stations, virtual museums, 
tourist information centres, etc. Most language learning Web sites at universities, language centres or 
international language organisations provide Web pages with links to these Web resources. Their 
webmasters index all these sites under different headings and sub-headings according to different 
languages and/or topics, e.g. grammar, media, culture, etc. Thus, these ‘collection-of-links’ pages are 
seen as launch pads to other sites on the Web, as they bring together different types of resources 
which can be accessed from one site. 

A collection of links is one way of concentrating the power of the Web as a teaching/leaming aid by 
centralising resources which might best address the diversity of learning needs which our students 
may have. At the same time it helps learners and teachers in their Web browsing, focusing their 
attention on useful resources without wasting too much time. Therefore, classification of Web 
resources allows for ease of reference and readiness of access, and helps indirectly to integrate the 
Web into the language curriculum. 

Though classification is essential, it is not in itself sufficient. It is true that the simple fact of selecting 
sites is an unconscious way of evaluating since we try to choose what we think can be of value for our 
own learners to be included on our respective Web pages. However, to date, on most of the Web 
pages which collect links to language-specific Web resources, there is no advice on the quality and 
usefulness of accessing the links provided. At most, they provide a summary of what one can find in 
following a link. What we propose here is a way of evaluating language-learning Web sites in a 
systematic and explicit way in order to enhance their pedagogical value. 

The Web has a lot to offer, but not all sources are equally valuable or even reliable. Most of the 
language resources on the Web are generally unfiltered, unlike most print materials which have been 
evaluated by an editorial team. A need, therefore, exists for the development of an evaluation 
procedure that will assist learners and teachers to assess the value of language leaming/teaching Web 
sites and for pedagogical and technical criteria to guide their future designs. 






3 




RGH WORKING PAPERS IN APPLIED LINGUISTICS, number 9 (1998) 



ISSN 0959-2253 



A project, called Ewe/mation, was established at the Institute for Applied Language Studies, 
University of Edinburgh, to develop such an evaluation procedure and to put it into practice using a 
Web page indexing Spanish language learning resources. As a consequence, a rating system has been 
developed to provide information which guides our learners in understanding the limitations or 
advantages of following a link. This rating system is in the form of stars placed next to a link. The 
user can click the star(s) and then see a detailed description of the evaluation results for that particular 
Web site. 

2. Preliminary considerations 

There are already services on the Web that evaluate and rate certain sites. Symbols and logos at the 
bottom of homepages are beginning to spring up saying “awarded 3 stars by Magellan”, “Top 5% of 
the Web”, etc. However, this is of limited value since most of the time we do not know the criteria 
they follow, and even when stated these criteria do not differentiate any subject domain or audience. 
To provide more valuable rating systems we should consider the context and focus on specific 
subjects, in our case CALL. All-purpose rating systems only produce a perceived value which in 
essence means nothing. Some search engines, however, include review sections by subject specialists 
in various areas of expertise, but their implementation of the evaluations is frequently superficial, 
subjective or biased. 

The problems of low quality and credibility of Web resources, and the need for coherent evaluation 
procedures, have already been suggested by several librarians (Alexander & Tate, 1997; Tillman, 
1997; Grassian, 1997) and by some educators (Schrock 1996; Kirk 1996; Wilkinson et al. 1997) who 
are trying to develop sets of criteria and evaluation surveys which will help learners and teachers to 
critically evaluate information on the Web. EweMiation is in line with these studies but specifically 
addressed to language learning. 

Ewe^uation should be understood here as formative evaluation (Rea-Dickins & Germaine, 1992:26), 
i.e. intended to identify strengths and weaknesses of language learning material on the Web. Such 
evaluation allows for raising awareness of the learning process in a formal way and thus for 
developing the evaluators’ critical thinking. This is an important skill to be developed due to the vast 
and varied information available on the Web. 



At present, there are still limitations on interaction using the Web for language learning. Bush (1996) 
states that the traditional HTML model is “limited in its ability to provide such simple interactions as 
answer judging as we often see in interactive material. This click and branch methodology is fine for 
accessing reference materials but is limited in its application to tutorials” (1996:2). However, Web 
technology is evolving dramatically and some materials are beginning to use program languages, such 
as CGI Scripts, JavaScript and Java, which allow a higher level of interactivity. It is true that the 
problem of speed of interaction due to limited bandwidth still remains: “the delays of learner’s click 
and the subsequent response (the system latency) can be disconcerting” (Bush, loc.cit:2), if not 
tedious, depending on the Web system used. New technologies will soon solve these problems and 
language teachers will then be able to make highly interactive Web materials. Therefore, parameters 
by which to judge Web resources will also change since the technology will have evolved and our 
expectations will be more demanding. 



Web technology has not yet reached the same level of maturity as multimedia technology, and so we 
must not fall into the trap of evaluating Web resources taking multimedia technology as the point of 
reference. Comparisons with other media cannot be the best way forward. What we need to judge is 
the medium itself as it is now, with all its limitations, in relation to the learner and the learning 
context. That said, however, evaluation procedures designed for printed material and CALL software 
were a source of inspiration in identifying possible evaluation criteria for language learning Web 
rces. 
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The Web provides a flexible medium for publishing language learning materials since it easily allows 
for revisions and updated versions. Through e-mail, for instance, an author can obtain direct feedback 
from learners and teachers on how they make use of his/her pages, and can improve them accordingly 
without going through the painful re-editing of printed material. In that sense, the very dynamic 
nature of the Web calls for formative evaluations, of which the Web author can easily take advantage. 

On the other hand, the fact that the Web is a continually evolving environment also poses problems in 
terms of the durability of the evaluations. For instance, once a site has been evaluated it has a “sell-by 
date” since the author could implement changes after receiving feedback from the users, so making 
the evaluation no longer valid; or even worse, an evaluation of a site might have been posted minutes 
before an updated version of that site is going online. This underlines the fact that evaluation of Web 
resources has to be understood as an ongoing process. 

Evaluation, in the educational setting, is sometimes perceived as a threatening procedure and always 
has the potential to be controversial. The main reason for this is that when evaluating we are judging 
against specific criteria which are based on the evaluators’ own approach to, in our case, CALL, and 
on their own learning context. These may not necessarily agree with one’s particular beliefs and 
situation. Thus, the evaluation framework stated below is determined by the context of the evaluation 
which is being implemented. However, with this in mind, we have attempted to identify criteria as 
comprehensible and universal as possible for the majority of CALL situations. 

3. Ejggfeuatjon framework 

The following framework is intended to outline the purpose and method of the evaluation process. 
This will allow anyone involved in the evaluation and anyone reading it to be clear about issues of 
why, what, who and how. 

3.1 Objectives 

The main objectives were: 

• to start a process which is desperately needed due to the wealth of information available, some 
of which is a “tremendous amount of garbage - sites that are redundant, disconnected, 
unprofessional, irrelevant” (Green 1997:256). 

• to make learners and teachers realise that Web resources need to be looked at critically. 

• to assist learners and teachers to evaluate their quality 
and ultimately: 

• to motivate discussion in the profession in order to improve the quality of the language 
resources on the Web. 

3.2 The evaluators 

Traditionally, evaluations of materials are carried out either by teachers, e.g. when selecting 
appropriate textbooks or software for their learners, or by students, e.g. at the end of the course. The 
Web allows for a more co-operative kind of evaluation where both learners and teachers can work 
together. 

In our case, both teachers’ and learners’ points of view are taken into consideration in order to give a 
coherent and balanced evaluation. Each party is actively engaged in the evaluation process more 
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or less at the same time. Both will answer the same questions so that it will make it easier to analyse 
them statistically. This does not mean that we are unaware that learners and teachers have different 
interests and different experiences. However, these will be reflected in their answers and thus possible 
differences spelled out. In that sense we could later study the reasons and consequences for those 
differences. Other positive reasons for involving students are that the evaluation procedure could 
develop learners’ critical thinking and reasoning skills, and may also be a means of increasing their 
awareness of what is useful for them as language learners and be, thereby, a form of learner training. 

IALS Modem Language learners are adult students with various professions, e.g. teachers, university 
students, lawyers, secretaries, bank clerks, etc. and usually highly motivated. They have a two-hour 
language session once a week for three terms, which means 66 hours per academic year. Age ranges 
from 18 to over 70. Students who took part in the project were volunteers from Beginners to 
Advanced, most of whom were computer-literate, with some experience in using the Web. 

3.3 The Web resources 

We are evaluating Spanish learning related Web sites indexed into the following sections: 

• Lessons: Sites designed to teach Spanish, mainly courses and tutorials. 

• Practice: Web sites designed to provide practice on different language skills or language 
systems. 

• Reference: i.e. dictionaries, grammars, vocabulary lists, etc. 

The difference between the first two categories is sometimes blurred since some courses are designed 
focusing mainly on one aspect of language learning. Besides, their main objective is the same: to 
provide an environment through which the learner can improve his/her language skills. The difference 
of classification is one of emphasis and structure. The first section covers sites more related to the 
traditional structure of a textbook, generally integrating different aspects of language learning, 
whereas the second includes sites designed specifically to tackle one aspect of language learning. 

3.4 Criteria 

What is it that makes a ‘good practice’ language learning Web site? The first step is to identify clearly 
the criteria by which we judge whether a Web site is very good, acceptable or bad. Any evaluation 
should be undertaken according to certain guiding principles using carefully defined criteria. 
However, when faced with a new medium, identifying the criteria is not an easy task since the 
designer and the user are inexperienced and still experimenting on how best to use it. 

One of the priorities when developing the criteria was to restrict these to the essential and the very 
general. It was perceived that identifying lots of detailed criteria was not going to assist the 
evaluators but confuse them. The different criteria identified for the evaluation are: 

CRITERION I : Quality and usefulness of the material 

This criterion refers to forming judgements about how valuable for one’s learning process the 
resource on the Web is. We need to consider to what extent the purpose of the site meets the learners’ 
needs. 
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CRITERION 2: Coverage 



This criterion deals with the depth and breadth of the content provided. The main issue to be 
considered is the level of detail the resource provides. This should be appropriate and balanced for the 
purpose of the Web site and its intended user. It should be made clear that more does not necessarily 
mean better. 

CRITERION 3: Userfriendliness 

The usability of a Web site is dependent on how well it is designed following basic instructional 
design standards. This criterion deals with the aesthetics of the design, the structure and organisation 
of the resource and the navigation of the content, i.e. how easily the resource can be explored. 

CRITERION 4: Motivating 

This criterion is concerned with how engaging a Web site is and thereby deals with the affective 
dimension of the learning process. 

CRITERION 5: Promoting active learning 

This criterion is related to the concept of interactivity. In the computer world interactivity is 
understood as simply clicking a link or selecting an option. In that sense, most resources on the Web 
are “interactive” in that the user can decide what to see and where to go. In an educational setting, 
however, the concept of interactivity should go beyond this. An educational interactive resource 
should require some sort of output on the part of the user. In that respect, our definition coincides 
with that of Lafford et ai , (1997:221) : “ interactive refers to an interactivity that requires oral or 
written production in the target language”. In order not to confuse the evaluators, the criterion was 
entitled “promoting active learning”, which means that learners should have an opportunity to receive 
input and produce output in the target language. This criterion is thought to be essential for the 
development of pedagogically sound language Web resources. 

CRITERION 6: Feedback 

This criterion is reserved for resources which provide tasks and/or exercises. It deals with the quality 
of the feedback provided, if any. 

Because of the language learning Web site categories’ differing functions, the Lessons and Practice 
section are evaluated following the 6 criteria, whereas the criteria assigned to sites in the Reference 
section are restricted to the first three. For instance, an online dictionary is rated by taking into 
account its quality, coverage and usability. Though some learners could find dictionaries motivating 
(criterion 4), it could be said that an online dictionary’s aim is not to motivate learners to learn a 
language, but simply one of support and reference. 

To assist the evaluators to clarify the meaning of each criterion and avoid possible misunderstandings, 
a handout with explanations was provided. Under each criterion open questions were outlined in order 
to make the evaluators reflect upon different aspects that characterise the corresponding criterion (see 
Appendix I). These questions serve as indicators of quality. 

3.5 Procedure 

The Ewefcuation project was conducted in five stages, as described below: 

The first stage was to draw up a list of Web sites that might be useful for our students. 
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The second stage was to classify them in a way that was clear for the learner (see 3.3), and to design 
the Web page with the links to these sites. 

The third stage was the preparation for the evaluation, i.e. thinking about and designing an evaluation 
procedure to rate language Web sites. The evaluative procedure chosen was a questionnaire which 
reflects the criteria stated. The evaluator had to rate each criterion on a 5 -point scale, from excellent 
to poor, for each site. This questionnaire was intended to be online but for technical reasons and due 
to time constraints it had to be administered using handouts (see Appendix II). 

The fourth stage was to collect the data. A special week was organised for volunteer students to come 
along and take part in the project. Some decided that they wanted to take part but preferred to do it in 
their own time since they had access to the Web. These learners returned answered questionnaires by 
post. Teachers were briefed about the project on an individual basis and evaluated the Web sites in 
their own time. To make the data relevant at this stage, evaluators were asked to concentrate on seven 
specific sites. Learners and teachers filled in the questionnaires and submitted their responses. 

The last stage was to process the data statistically in order to rate the resources. This process resulted 
in the awarding of a number of stars to the different Web sites evaluated: ranging from five for an 
excellent Web site to one for poor Web sites. The overall score is obtained by averaging the learners’ 
and the teachers’ scores on each criterion. To see details of the score awarded to a site, a user can 
click on the star(s) next to the link and will obtain the averages for each criterion together with the 
teachers’ and learners’ respective comments. 

As mentioned before, stages 4 and 5 were intended to be carried out online. In future, the evaluators 
will complete the questionnaire and submit it online and the data will be processed automatically. 
Some CGI programs using Perl 5.0 have been designed in order to achieve this. 

3.5.1 Technical features 

As far as programming is concerned, the site is divided into three main interconnected areas: 

• Output Area: 

The evaluation Web pages are generated by CGI Scripts. These pages are created with stars which 
rate the quality of the sites provided. Input via the questionnaire dynamically updates these Web 
pages. 



• Input Area: 

The questionnaire information is entered by the evaluators into a Web form which uses CGI Scripts to 
place the data in the associated databases. This information is later used by the output section to form 
the rating of the evaluated sites and by the administration section from which the data can be edited. 

• Administration Area: 



This section is divided into three parts: one that allows the addition of a new link, creating a site 
database (or adding to an existing one), and the second where the CGI scripts search the data in the 
databases presenting it in a form for editing and manipulating. 

The last part involves security and has two aspects. The first is that the site is protected by passwords, 
one for the students, another for the teachers and also one for the administrator. This security measure 
stops (intentional or accidental) changes to ‘volatile’ data, e.g. a student inadvertently deleting a site 
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in the administration area. The second part entails checking all the links in the databases for broken, 
missing or moved linked pages. 

The technical solutions to some of the problems that arise from the complexity of evaluating a Web 
site are shown below: 



Problem 


Technical solution 


Only one evaluation is not representative 
enough 


Evaluation is counted but stars are not 
shown in total score. 

Minimum of 5 students and 3 teachers 


Outdated evaluations 


User can see when the evaluation was 
carried out 


Radical modification of a Web site 


Send a message alerting to changes 


URL moved 


Send a message alerting that URL moved 


Hacking 


Password protected 


Different ratings to be considered: 

6 criteria 

5 criteria: Sites without feedback 
3 criteria: Reference sites 


Identify differences and apply statistical 
analysis accordingly. 


Learners’ level biased 


Indication of level of learners on the 
comments section 



4. Evaluating Ewgfruation 



This project has, to a large extent, met successfully the first three objectives stated in 3.1. Five 
teachers took part in the evaluation, most of whom completed all the questionnaires. They have a lot 
of experience as language teachers but had only basic knowledge about the Web as a resource for 
language learning. This project helped them to realise the potential of the Web as a leaming/teaching 
tool. 

Learners’ response was fairly positive. Out of 300 students of Spanish, 70 initially expressed interest 
in taking part. 31 came to the evaluation week to take part in the project and 25 asked for the 
questionnaires to be sent, 12 of whom sent them back. Some students managed to evaluate the seven 
sites requested, others just evaluated as many as possible in the time they had available. 

None of the students or teachers reported difficulty in understanding the criteria, and all thought it 
was an excellent idea of which they could take advantage. Most of them praised the clear design of 
the Web pages. 

Problems of full functionality of some sites encountered by some evaluators at certain times forced 
their evaluations of those sites to be postponed. Learners at home who did not have multimedia 
facilities could not fully evaluate some of the sites. 



5. Conclusion and future possibilities 



In general terms the positive feedback received so far confirms that at least we are starting off on the 
right foot. There are still problems to be solved and aspects to be improved. The criteria stated in the 
framework, for instance, should be validated by an external panel of evaluators so that they could be 
applied to a more extensive context. These could, then, be reviewed and re-adjusted, e g. add more 
criteria not accounted for or simply more indicators under each criterion. Evveimation is seen as just 
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the beginning of an ongoing project that is much needed by language learners and teachers world- 
wide. 

In that respect, other universities may be interested in adopting a similar model, and some national or 
international organisations, such as the CTI in the UK or EUROCALL, may consider a project of this 
kind on a more universal level. What is clear is that we cannot assist our learners and teachers by just 
simply providing a collection of links of what is available on the Web. This could be easily achieved 
by the numerous search engines already available. Language centres, university departments and 
international organisations, apart from providing advice on how to exploit the Web, should offer 
further specific information in the form of reviews or rating systems that reflect ‘good practice’ 
language learning Web sites. 

I hope that the evaluation model presented here will stimulate constructive discussion and have a 
positive washback effect which ultimately may contribute to the improvement of the present Web 
sites and possibly the development of a set of guidelines for the design of future quality language 
learning Web sites. 



Notes 

1 . This paper was presented at the EUROCALL 97 Conference in Dublin. 
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APPENDIX 1 



EVALUATING SPANISH WEBSITES 
Instructions on How to Fill the Questionnaire 



The following are some of the points you have to consider before you fill in each category. 

Remember to answer all the questions for the sections Lessons and Practice, taking into account that 
question 6 is only appropriate when there are exercises to do on the website. 

For the section Reference you ONLY have to answer the first 3 questions. 



Criteria and Questions: 

1. Quality and usefulness of the material: 

• Does the material provide a valuable aid to language learning? 

• Is the information appropriate for what you expect to find on the website? 

• Is there useful information to make the site worthwhile visiting? 

2. Coverage 

• Is there enough information provided? 

• Is there so much that it is impossible to assimilate or manage? 

Dictionary : did you find the words you looked up or is it somehow limited? 

3. User-friendliness 

• Design: Is the design of the website attractive? 

• Instructions: Are the instructions clear and comprehensible? 

• Organisation: Is the website well-organised? 

• Navigation: Can you move from page to page without getting lost or confused? 

• Speed: Does the page downloads efficiently? 

4. Motivating 

Does the website: 

• present material in a way that holds attention? 

• stimulate interest in the subject? 

• encourage learning and study? 
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5. Promoting active learning 

Does the website ask or prompt the user to: 

• do something (i.e. do some exercises, reply...)? 

• take part in some way? 

• interact with somebody else (i.e. author or other students)? 

6. Feedback (only when there are exercises) 

• Is the feedback provided helpful? 

• Is a more clear or in-depth explanation necessary? 



You may make any additional comments about the website in the space provided. 

Remember that if you feel that you are unable to evaluate a website for any reason you can indicate 
this on the questionnaire sheet. 

If you have any problem, e-mail: J.Pujola@ed.ac.uk 

Thank you for your co-operation. 
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APPENDIX II 

EwiAuation Questionnaire - Students / Teachers* 

This form gives you an opportunity to indicate your opinion to the learning value of the Spanish links 
provided. Please consider each question separately, without letting your overall impression to the 
material blind you to particular areas of the site's strengths or weaknesses. For each question, choose 
the response that best reflects your personal opinion. 

Please select the name of the site being evaluated here: 





excellent 


very good 


satisfactory 


poor 


very poor 


1 . Quality and usefulness of the material: 












2. Coverage: 












3. Userfriendliness: 












4. Motivating: 












5. Promoting active learning: 












6. Feedback: (only when there are EXERCISES) 













Please add any comments you might have: 



□ I was unable to evaluate the site because 



Which level are you studying at IALS? 



* N.B. The teachers’ questionnaire was the same as learners’ except for the section after the 
comments which only included the following question: 



Which level(s) at IALS would you recommend this site for? 
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